Software testing takes time, and is a necessary element of the software development process. The quality of a software product is typically measured in terms of how well it is tested by a test suite and how few problems or bugs (unintended behavior of software) are found after release.
There are different types of testing methods available to determine software quality by applying a test suite to a software product. Test suites are pre-defined groups of tests designed to make sure certain aspects of a software product are tested. Various groups might define test suites for different purposes, for example, one or more of a customer, research and development engineer, or coder may develop a test suite for software.
One measure of software quality, called line coverage, measures the percentage of the code within the software product that is executed and tested through the associated test suite. For example, 10% line coverage means that 10% of the lines of code that make up the software product are executed when applying the test suite. Conversely, 90% of the lines of code that make up the software product are not executed when applying the test suite when the line coverage is 10% A similar metric of software quality, called functional coverage, measures the percentage of functions and/or subroutines that are executed and tested through the associated test suite. For example, 20% function coverage means that 20% of the functions present in the software have been called from at least one test in the test suite. These and similar metrics provide some basis for software code, but fail to give a full picture of behavior of the product. Whether or not a function is called does not indicate whether it was called correctly and whether or not it functioned correctly when called. Similarly, it does not give insight into whether or not variations in the call of the function impact software product quality.
Another type of testing method, called black box testing, measures the quality of software by testing the product behavior based on a requirements document and assuming no knowledge of the code of the software product. However, black box testing is time consuming. For black box testing and other similar metrics, it is nearly impossible to ensure that all updates and scenarios are tested. And, similarly to the above metrics, does not give insight into whether or not variations of the inputs will impact software product quality.
Despite of all the quality tools being used extensively during the software product development phase, production software (a software product that has been released) ends up with many bugs. These bugs result in additional technical support required for the software product, redevelopment iterations to correct the software product, and increased costs associated with software product development. Additionally, software product updates may introduce additional bugs. These additional bugs may occur at various product events, such as when a product is changed, for example, with a new version or release; between phases of product development such as alpha release, beta release, and gold release. Additional bugs may be introduced when changes are made to portions of the software product, for example, when changes are checked-in to a code management system, or associated with a validation schedule, for example weekly or at a milestone. Therefore, product development that may introduce bugs into the product code may happen in a variety of stages, both pre- and post-release of the software.
In addition to the bugs that may be introduced despite adherence to a rigorous software product development and quality assurance process, the reality of compressed development and software product update schedules associated with modern software development may require a timeframe that results in skipping many steps of a quality assurance process. For example, software products may be developed or updated by someone other than the standard product and development team. For example, updates may be coded by research and development teams creating functionality or fixing complex bugs and then released to a customer under a tight time frame that allows minimal formal, if any, testing. As another example, multiple features may be added to a software product in parallel on a schedule that does not allow complete testing for compatibility and interaction between those multiple features before release. Different features may be implemented by different groups in diverse locations and time zones, impairing the ability of teams working on the same software product to align on development. The likelihood of even more bugs introduced into the software is increased when not following best practices in software product development.
Despite the relative maturity of the software industry, with decades of software testing using these methodologies, software quality issues persist, costing time and money when problems are exposed after release. Metrics similar to line coverage, functional coverage, and black box testing are not producing bug-free or bug-minimized releases. Additionally, existing metrics do not measures issues associated with customer usage. Customer issues are found when a software product is shipped to a customer and the customer finds a bug in the software. It would be desirable to have a quality metric that includes customer usage issues in the metrics for software quality testing, to proactively minimize bugs.
In spite of long-felt need for a more robust testing environment, both in a robust development process or an abbreviated one, at this time there has not been a systemic approach to ensuring that testing truly covers so many of the scenarios likely encountered by the customer or end-user in production and other software products.